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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



CLAIMS 



[Claim(s)] 

[Claim 1]While photoing a landscape image near [ in an experiencing person's sight line 
direction ] a planned construction site, Send an experiencing persons position information and 
sight line direction information to a control means, and computer graphic model data of a 
structure buitt in the above-mentioned planned construction site is changed into image data 
corresponding to an experiencing person's position and a sight line direction, Create a computer 
graphical image of the above-mentioned structure which masked a background, and distance 
from an experiencing person's position to a structure is calculated in a computer graphic of the 
above-mentioned structure after that, Distance to a realistic object in a scene corresponding in 
a computer graphic of the above-mentioned structure is compared with distance to the above- 
mentioned structure, A field judged that the above-mentioned realistic object exists in the 
experiencing person side from a construction predetermined position of a structure in a field of a 
computer graphic of the above-mentioned structure is masked, A method for carrying out the 
virtual experience of the scene in the state where a structure was built in a planned 
construction site characterized by treating as transparent a portion masked in a computer 
graphical image of the above-mentioned structure, compounding the above-mentioned landscape 
image, and making it display. 

[Claim 2]An image photographing means for photoing a landscape image near [ in an experiencing 
person's sight line direction ] a planned construction site, A position detecting means for 
detecting an experiencing person's position information, and a direction finding system for 
detecting an experiencing person's sight line direction information, A depth detection means for 
detecting distance from an experiencing person to a realistic object in the above-mentioned 
scene, Computer graphic model data of a structure built in the above-mentioned planned 
construction site based on an experiencing person's position information and sight line direction 
information is changed into image data corresponding to an experiencing person's positron and a 
sight line direction, While creating a computer graphical image of the above-mentioned structure 
which masked a background, In a computer graphic of the above-mentioned structure, distance 
from an experiencing person's position to a structure is calculated, Input distance to a realistic 
object in a scene corresponding in a computer graphic of the above-mentioned structure from 
the above-mentioned depth detection means, and distance to the realistic object concerned is 
compared with distance to the above-mentioned structure, A control means which creates a 
computer graphical image of a structure which masked a field judged that the above-mentioned 
realistic object exists in the experiencing person side from a construction predetermined position 
of a structure in a field of a computer graphic of the above-mentioned structure, An image 
compositing means which treats as transparent a portion masked in a computer graphical image 
of the above-mentioned structure, and compounds the above-mentioned landscape image, A 
device for carrying out the virtual experience of the scene in the state where a structure was 
built in a planned construction site by which it is having-displaying means which displays image 
composing compounded by this image compositing means characterized. 

[Claim 3]A method according to claim 1 or 2 or a device, wherein the above-mentioned control 
means changes this shadow while giving a shadow to a computer graphic of a structure based on 
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date information, 

[Claim 4]A method according to claim 1, 2 t or 3 or a device, wherein the above-mentioned 
control means changes these shade and shadow while attaching shade and shadow to a 
computer graphic of a structure based on illuminance information. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DETAILED DESCRIPTION 

[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]When this invention performs scene examination in environmental 
assessment, etc., it relates to the method or device which could be made to carry out the virtual 
experience of the scene in the state where the structure was buiit in the planned construction 
site, a priori there. 
[0002] 

[Description of the Prior Art]Usually, when scene examination was performed, computer graphic 
(henceforth CG) animation, a photo montage, etc. were used, and a screen (video) and printed 
matter were performing. 
[0003] 

[Problem(s) to be Solved by the Invention]However, conventionally, since it was not directly 
carried out at the spot when [ almost all ] scene examination was performed, the virtual 
experience (viewing) of the scene of the state after the structure was built at the spot which 
performs scene examination could not be carried out directly, and realistic examination was not 
able to be performed. 
[0004] 

[Means for Solving the Problem]While a method of this invention photos a landscape image near 
[ in an experiencing person s sight line direction ] a planned construction site, Send an 
experiencing person's position information and sight line direction information to a control means, 
and CG model data of a structure built in the above-mentioned planned construction site is 
changed into image data corresponding to an experiencing person's position and a sight line 
direction, Create CG image of the above-mentioned structure which masked a background, and 
distance from an experiencing person 's position to a structure is calculated in CG of the above- 
mentioned structure after that, Distance to a realistic object in a scene corresponding in CG of 
the above-mentioned structure is compared with distance to the above-mentioned structure, In 
a field of CG of the above-mentioned structure, a portion which masked a field judged that the 
above-mentioned realistic object exists in the experiencing person side from a construction 
predetermined position of a structure, and was masked in CG image of the above-mentioned 
structure is treated as transparent, the above-mentioned landscape image is compounded, and it 
was made to display. An image photographing means for a device of this invention to photo a 
landscape image near [ in an experiencing person's sight line direction ] a planned construction 
site, A position detecting means for detecting an experiencing person's position information, and 
a direction finding system for detecting an experiencing person's sight line direction information, 
A depth detection means for detecting distance from an experiencing person to a realistic object 
in the above-mentioned scene, CG model data of a structure buiit in the above-mentioned 
planned construction site based on an experiencing person's position information and sight line 
direction information is changed into image data corresponding to an experiencing person's 
position and a sight line direction, While creating CG image of the above-mentioned structure 
which masked a background, In CG of the above-mentioned structure, distance from an 
experiencing person's position to a structure is calculated, Input distance to a realistic object in 
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a scene corresponding in CG of the above-mentioned structure from the above-mentioned 
depth detection means, and distance to the realistic object concerned is compared with distance 
to the above-mentioned structure, A control means which creates CG image of a structure 
which masked a field judged that the above-mentioned realistic object exists in the experiencing 
person side from a construction predetermined position of a structure in a field of CG of the 
above-mentioned structure, An image compositing means which treats as transparent a portion 
masked in CG image of the above-mentioned structure, and compounds the above-mentioned 
landscape image, It had a displaying means which displays image composing compounded by this 
image compositing means. According to a method and a device of this invention, a scene before 
and behind a structure which is due to be built, and this structure is compoundable with reality. 
[0005] 

[Embodiment of the Invention] First, the outline of the virtual experience device by the 
embodiment of the invention 1 is explained based on dr awing 1 . 1 is an image photographing 
apparatus for always photoing the landscape image near [ in an experiencing person's sight line 
direction ] a planned construction site, for example, the camera using a CCD element is used. 2 
is a position sensing device for always detecting an experiencing person s three-dimensional 
position information (X, Y, Z data), for example, GPS devices, such as RTK(real-time kinematic)- 
GPS and D(differential)-GPS, are used. 3 is a direction finding system for always detecting an 
experiencing person's sight line direction information (an experiencing person's look azimuth data 
and look elevation angle data), For example, an acceleration sensor, what detects a sight line 
direction combining a gravity sensor, the magnetic sensor which detects a sight line direction 
using a magnetic field (magnetic field), etc. are used. The non prism range finder (laser scanning 
device) which 4 is a depth sensing device which detects the depth information (distance data 
from an experiencing person to each realistic object in a scene) of the realistic object in the 
scene which an experiencing person views, for example, measures distance by a reflected wave 
is used. What is necessary is just to be able to measure this non prism range finder by directing 
the candidate for measurement, even if there is no target although the direction of what is 
measured using a target is accurate. 5 is a personal computer (henceforth a personal computer) 
as a control device, and structure image display software is incorporated. The three-dimensional 
CG model data of the structure built in the planned construction site in a virtual experience 
person s sight line direction is beforehand inputted into this personal computer 5. As model data, 
CG pattern of various structures which are due to be built, the three-dimensional position 
information on a structure, etc. are inputted beforehand. The chroma keyer which 6 is an image 
compositing device, for example, treats as transparent the color of the range which inputted two 
video signals and was specified among one video signal, and compounds it to the 2nd video signal 
is used, 7 is a display which displays the picture combined by the image compositing device 6, for 
example, HMD (head mount display device) is used. The image signal converter from which 1a 
changes a picture signal into a video signal, the controller by which 3a controls the output timing 
of sight line direction information to the personal computer 5, The controller by which 4a 
controls the output timing of depth information to the personal computer 5, the down converter 
from which 5a changes the data of CG into a video signal, and 6a are display controllers which 
control the display 7. The cable is made to perform the signal transmission of a picture signal 
and a sight line direction, a position, and depth information (data). The devices 1, 2, 3, 4, and 7 
are attached to an experiencing person's head. 

[0006] Next, operation of a virtual experience device is explained based on dr awing 2 - drawin g 6. 

(1) As shown in drawing 2, always acquire the landscape image A of an experiencing persons 
sight line direction with the image photographing apparatus 1. 

(2) The three-dimensional position information of the experiencing person always acquired with 
the position sensing device 2 and an experiencing person's sight line direction information always 
acquired with the direction finding system 3 are simultaneously sent to the personal computer 5. 
Structure image display software changes into the image data corresponding to the experiencing 
person's position and sight line direction the CG model data of the structure which is due to be 
built based on these information, and displays it on the display 7. For example, as shown in 
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drawing 3, the CG image B of the tank T as a structure which is due to be built is created, and it 
displays on real time (when an experiencing person's position and sight line direction change, the 
size and display position of CG change). In this case, there is the necessity (it resets) of 
coinciding the starting point of the CG image B and the starting point of the position sensing 
device 2, and deciding them beforehand. The background C of the CG image B is used as 
masking. That is, in order to treat as behind transparent, it distinguishes by a different color from 
the color of CG of the tank T (for example, if the color of CG of the tank T is gray, the 
background will be made blue and will treat this blue portion as transparent behind). 
(3) Next, by structure image display software, calculate the outline of CG of the above- 
mentioned tank T, and calculate the context of a background and CG to the field in this outline. 
That is, in CG of the above-mentioned tank T, the distance from an experiencing persons 
position to the outline of the tank which is a construction schedule is calculated. And the 
distance data to the realistic object in a scene corresponding with the depth sensing device 4 in 
the outline of CG of the above-mentioned tank T is acquired, this distance data is sent to the 
personal computer 5, and it compares with the distance to the above-mentioned outline. The 
front face D which assumed the case where the experiencing person I looked at the tank T 
which is a actual construction schedule will be displayed, and CG of the tank T will call it the 
distance from the experiencing person I to each portion of the front face D of the above with the 
distance to the above-mentioned outline, as shown in drawing 4 . That is, in the personal 
computer 5, the distance to the front face D of the tank of the imagination which is in agreement 
with an experiencing persons sight line direction F is compared with the distance of the realistic 
object located in the front or back from this front face D, and the context is grasped. For 
example, if the realistic object in which the realistic object located ahead of the tank T which is 
a actual construction schedule is located in tree JF and back is the tree JB and the personal 
computer 5 compares the distance to Xa, Fa, Xb, and Fb of drawing 4 , it turns out that tree JF 
which is a realistic object is located ahead. If the personal computer 5 compares the distance to 
Xa, Ba f Xb, and Bb, it turns out that the tree JB which is a realistic object is located back. 
Actual scene within the limits which controls the measurement direction of the depth sensing 
device 4 based on the model data (three-dimensional position information) of CG of the tank T, 
and specifically corresponds in CG field of the tank T is made to scan sequentially. That is, the 
distance data to the actual realistic object equivalent to the position of each pixel of CG is 
incorporated one by one, and the successive approximation is carried out to the corresponding 
distance data to each portion of the front face D of CG. And CG (same blue as CG background 
was used) image which masked the CG field X of the tank T judged that the above-mentioned 
realistic object exists in a near side (experiencing person I side) from the construction 
predetermined position of a tank in the field of CG of the above-mentioned tank T is created 
(dra wing 5 ). 

The landscape image A of an experiencing person's sight line direction acquired by (4) and (1) 
( drawing 2 ) and CG image (d rawing 5 ) to which masking was added by (3) are compounded. That 
is, by the image compositing device 6, the masking portion (blue portion) of CG image ( drawing 5) 
is treated as transparent, the landscape image A is compounded, and the image composing Z like 
drawing 6 is displayed on the display 7. Namely, the object (the left-hand side of d rawing 2 and 
the building a of 5 part, the building b of drawing 2 ) located behind the tank T hides by CG of the 
tank T, The image composing Z compounded so that the object (dra wing 2 , the building c of 5, 
the trees d) located in the near side of the tank T might hide a part of CG of the tank T will be 
displayed on the display 7. 

[0007]Therefore, the scene before and behind the structure which is due to be built, and this 
structure can be compounded with reality, and it comes to be able to perform a real virtual 
experience (viewing). Since he is trying to display continuously the picture ( drawi ng 2, 3, 5, 6) 
which processed above-mentioned (1) - (4) to the display 7 of the experiencing person head on 
real time, it comes to be able to perform the virtual experience which has a real feeling there. 
Since it ends with comparison of only a realistic object located in actual corresponding in outline 
of CG scene within the limits when performing distance comparison of the realistic object in a 
scene, and CG, it is not necessary to perform distance measurement of the whole scene, 
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distance measurement and the throughput of distance comparison are reduced, and processing 
efficiency is good. 

[0008]While calculating the position of light sources, such as the sun, and giving the shadow S1 
to CG of a structure by inputting date information into the personal computer 5 based on the 
position of said light source, If it is made to change the shadow S1 with change of time, a real 
feeling can be taken out to CG of a structure and CG can be brought close to the actual thing. 
While attaching shade and shadow (shade of a shadow) also in the portion to which light strikes 
upon CG of the structure in consideration of the surroundings lump of light by acquiring 
illuminance information with an illuminometer etc. and inputting this illuminance information into 
the personal computer 5 f If it is made to change these shade and shadow with change of 
illumination, a real feeling can be taken out by CG of a structure and CG of a structure can be 
brought more close to the actual thing. In this case, what is necessary is to input beforehand the 
model data of a shadow or shade and shadow to various structures into the personal computer 
5, and just to process change of the shadow by change of date information and illuminance 
information, or shade and shadow by software. Since a virtual experience can be carried out still 
with reality according to the above, still more realistic examination can be performed. 
[0009]Although the data of a picture signal and a sight line direction, a position, etc. is always 
acquired and he is trying to display a picture on the display 7 continuously above in real time, 
when required, data is acquired once and, of course, it may be made to display a picture each 
time. 

[0010]Without forming the depth sensing device 4 for exclusive use, as shown in drawi.ng_7 v the 
two image photographing apparatus 1A and 1B may be prepared, and it may constitute so that 
the azimuth difference of two pictures may detect depth. In this case, the depth calculation 
software by image processing is built into the personal computer 5A. 

[001 1]As the position sensing device 2 f a followed type total station other than a GPS device, 
etc. can also be used. A marker etc. are fixed to the described image photographing instrument 1 
or the display 7, and it may be made for calculation to detect the present virtual experience 
person's sight line direction from the physical relationship of the marker within a picture by 
reading into the personal computer 5 by making it into picture information, without using the 
direction finding system 3 for exclusive use. As the image compositing device 6, use TEROPPA 
which treats the specified color (monochrome) as transparent besides a chroma keyer, and 
compounds CG of the personal computer 5 to a video signal, or, Or it may be made to combine 
two pictures by the same processing as a chroma keyer or TEROPPA using the video input 
output function added to the personal computer 5. As the display 7, a notebook sized personal 
computer etc. may be used as FMD (face mount display device) or a movable display besides 
HMD. It may be made to transmit a picture and a direction, a position, and depth information 
(data) by radio other than transmission by a cable as a signal-transmission method. It may be 
made to transmit the above-mentioned data digitally using cable LAN or wireless LAN. It is not 
necessary to necessarily attach the image photographing apparatus 1 (1 A, 1B), the direction 
finding system 3, the position sensing device 2, the depth sensing device 4, and the display 7 to 
a head. For example, these may all be made to unite with a personal computer, and it is good 
only also considering an image photographing apparatus as a different body. What is necessary is 
in short, just to be able to detect direction of an image photographing apparatus. A CCD camera 
etc. may be attached to a telescope or binoculars and an image photographing apparatus may be 
constituted. 
[0012] 

[Effect of the InventionjAs mentioned above, according to this invention, at the spot of a 
planned construction site, since the virtual experience (viewing) of the scene in the state where 
the structure was built can be carried out directly and the virtual experience of the scene before 
and behind a structure can be carried out with reality, realistic examination can be performed. 
Since it is not necessary to perform distance measurement of the whole scene, distance 
measurement and the throughput of distance comparison are reduced and processing efficiency 
is good. While giving a shadow to CG of a structure based on date information, making it change 
this shadow or attaching shade and shadow to CG of a structure based on illuminance 
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information, Since a virtual experience can be carried out still with reality by making it change 
these shade and shadow, still more realistic examination can be performed. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word whrch can not be translated. 
3.1n the drawings, any words are not translated. 



TECHNICAL FIELD 



[Field of the InventionjWhen this invention performs scene examination in environmental 
assessment, etc, it relates to the method or device which could be made to carry out the virtual 
experience of the scene in the state where the structure was built in the planned construction 
site, a priori there. 



[Translation done.] 



http://www4.ipdlJnpit.go.jp/cgi-bin/tran_web_cgi„ejje?atw_u=http%3A%2F%2Fwww4,i... 2008/03/25 



JP,2001-101252,A [PRIOR ART] 



1/1 ^— V 



* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



PRIOR ART 



[Description of the Prior Art]Usually, when scene examination was performed, computer graphic 
(henceforth CG) animation, a photo montage, etc. were used, and a screen (video) and printed 
matter were performing. 



[Translation done,] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



EFFECT OF THE INVENTION 



[Effect of the Invention]As mentioned above, in this invention, at the spot of a planned 
construction site, the virtual experience (viewing) of the scene in the state where the structure 
was built can be carried out directly, and the virtual experience of the scene before and behind a 
structure can be carried out with reality. 
Therefore, realistic examination can be performed. 

Since it is not necessary to perform distance measurement of the whole scene, distance 
measurement and the throughput of distance comparison are reduced and processing efficiency 
is good. While giving a shadow to CG of a structure based on date information, making it change 
this shadow or attaching shade and shadow to CG of a structure based on illuminance 
information, Since a virtual experience can be carried out still with reality by making it change 
these shade and shadow, still more realistic examination can be performed. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

"LThis document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated 
3. In the drawings, any words are not translated. 



TECHNICAL PROBLEM 



[Problem(s) to be Solved by the Invention] However, conventionally, since it was not directly 
carried out at the spot when [ almost all ] scene examination was performed, the virtual 
experience (viewing) of the scene of the state after the structure was built at the spot which 
performs scene examination could not be carried out directly, and realistic examination was not 
able to be performed. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation, 

1This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



MEANS 



[Means for Solving the Problem]While a method of this invention photos a landscape image near 
[ in an experiencing person's sight line direction ] a planned construction site, Send an 
experiencing person's position information and sight line direction information to a control means, 
and CG model data of a structure built in the above-mentioned planned construction site is 
changed into image data corresponding to an experiencing person's position and a sight line 
direction, Create CG image of the above-mentioned structure which masked a background, and 
distance from an experiencing person's position to a structure is calculated in CG of the above- 
mentioned structure after that, Distance to a realistic object in a scene corresponding in CG of 
the above-mentioned structure is compared with distance to the above-mentioned structure, In 
a field of CG of the above-mentioned structure, a portion which masked a field judged that the 
above-mentioned realistic object exists in the experiencing person side from a construction 
predetermined position of a structure, and was masked in CG image of the above-mentioned 
structure is treated as transparent, the above-mentioned landscape image is compounded, and it 
was made to display. An image photographing means for a device of this invention to photo a 
landscape image near [ in an experiencing person's sight line direction ] a planned construction 
site, A position detecting means for detecting an experiencing person's position information, and 
a direction finding system for detecting an experiencing person's sight line direction information, 
A depth detection means for detecting distance from an experiencing person to a realistic object 
in the above-mentioned scene, CG model data of a structure built in the above-mentioned 
planned construction site based on an experiencing person's position information and sight line 
direction information is changed into image data corresponding to an experiencing person's 
position and a sight line direction, While creating CG image of the above-mentioned structure 
which masked a background, In CG of the above-mentioned structure, distance from an 
experiencing person's position to a structure is calculated, Input distance to a realistic object in 
a scene corresponding in CG of the above-mentioned structure from the above-mentioned 
depth detection means, and distance to the realistic object concerned is compared with distance 
to the above-mentioned structure, A control means which creates CG image of a structure 
which masked a field judged that the above-mentioned realistic object exists in the experiencing 
person side from a construction predetermined position of a structure in a field of CG of the 
above-mentioned structure. An image compositing means which treats as transparent a portion 
masked in CG image of the above-mentioned structure, and compounds the above-mentioned 
landscape image, It had a displaying means which displays image composing compounded by this 
image compositing means. According to a method and a device of this invention, a scene before 
and behind a structure which is due to be built, and this structure is compoundable with reality. 
[0005] 

[Embodiment of the Invention] First, the outline of the virtual experience device by the 
embodiment of the invention 1 is explained based on drawing 1 . 1 is an image photographing 
apparatus for always photoing the landscape image near [ in an experiencing person's sight line 
direction ] a planned construction site, for example, the camera using a CCD element is used, 2 
is a position sensing device for always detecting an experiencing person's three-dimensional 
position information (X, Y, Z data), for example, GPS devices, such as RTK(real-time kinematic)- 
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GPS and D(differential)-GPS, are used. 3 is a direction finding system for always detecting an 
experiencing person's sight line direction information (an experiencing person's look azimuth data 
and look elevation angle data), For example, an acceleration sensor, what detects a sight line 
direction combining a gravity sensor, the magnetic sensor which detects a sight line direction 
using a magnetic field (magnetic field), etc, are used. The non prism range finder (laser scanning 
device) which 4 is a depth sensing device which detects the depth information (distance data 
from an experiencing person to each realistic object in a scene) of the realistic object in the 
scene which an experiencing person views, for example, measures distance by a reflected wave 
is used. What is necessary is just to be able to measure this non prism range finder by directing 
the candidate for measurement, even if there is no target although the direction of what is 
measured using a target is accurate. 5 is a personal computer (henceforth a personal computer) 
as a control device, and structure image display software is incorporated. The three-dimensional 
GG model data of the structure built in the planned construction site in a virtual experience 
person s sight line direction is beforehand inputted into this personal computer 5. As model data, 
CG pattern of various structures which are due to be built, the three-dimensional position 
information on a structure, etc, are inputted beforehand. The chroma keyer which 6 is an image 
compositing device, for example, treats as transparent the color of the range which inputted two 
video signals and was specified among one video signal, and compounds it to the 2nd video signal 
is used. 7 is a display which displays the picture combined by the image compositing device 6, for 
example, HMD (head mount display device) is used. The image signal converter from which 1a 
changes a picture signal into a video signal, the controller by which 3a controls the output timing 
of sight line direction information to the personal computer 5, The controller by which 4a 
controls the output timing of depth information to the personal computer 5, the down converter 
from which 5a changes the data of CG into a video signal, and 6a are display controllers which 
control the display 7. The cable is made to perform the signal transmission of a picture signal 
and a sight line direction, a position, and depth information (data). The devices 1, 2, 3, 4, and 7 
are attached to an experiencing person's head. 

[0006]Next, operation of a virtual experience device is explained based on drawing 2 - drawing 6. 

(1) As shown in drawing 2 , always acquire the landscape image A of an experiencing person's 
sight line direction with the image photographing apparatus 1. 

(2) The three-dimensional position information of the experiencing person always acquired with 
the position sensing device 2 and an experiencing person's sight line direction information always 
acquired with the direction finding system 3 are simultaneously sent to the personal computer 5. 
Structure image display software changes into the image data corresponding to the experiencing 
person's position and sight line direction the CG model data of the structure which is due to be 
built based on these information, and displays it on the display 7. For example, as shown in 
drawing 3 , the CG image B of the tank T as a structure which is due to be built is created, and it 
displays on real time (when an experiencing person's position and sight line direction change, the 
size and display position of CG change). In this case, there is the necessity (it resets) of 
coinciding the starting point of the CG image B and the starting point of the position sensing 
device 2, and deciding them beforehand. The background C of the CG image B is used as 
masking. That is, in order to treat as behind transparent, it distinguishes by a different color from 
the color of CG of the tank T (for example, if the color of CG of the tank T is gray, the 
background will be made blue and will treat this blue portion as transparent behind). 

(3) Next, by structure image display software, calculate the outline of CG of the above- 
mentioned tank T. and calculate the context of a background and CG to the field in this outline. 
That is, in CG of the above— mentioned tank T, the distance from an experiencing person's 
position to the outline of the tank which is a construction schedule is calculated. And the 
distance data to the realistic object in a scene corresponding with the depth sensing device 4 in 
the outline of CG of the above-mentioned tank T is acquired, this distance data is sent to the 
personal computer 5, and it compares with the distance to the above-mentioned outline. The 
front face D which assumed the case where the experiencing person ! looked at the tank T 
which is a actual construction schedule will be displayed, and CG of the tank T will call it the 
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distance from the experiencing person I to each portion of the front face D of the above with the 
distance to the above-mentioned outline, as shown in drawing 4 . That is T in the personal 
computer 5, the distance to the front face D of the tank of the imagination which is in agreement 
with an experiencing person's sight line direction F is compared with the distance of the realistic 
object located in the front or back from this front face D, and the context is grasped. For 
example, if the realistic object in which the realistic object located ahead of the tank T which is 
a actual construction schedule is located in tree JF and back is the tree JB and the personal 
computer 5 compares the distance to Xa t Fa, Xb, and Fb of drawing 4 , it turns out that tree JF 
which is a realistic object is located ahead. If the personal computer 5 compares the distance to 
Xa, Ba, Xb, and Bb, it turns out that the tree JB which is a realistic object is located back. 
Actual scene within the limits which controls the measurement direction of the depth sensing 
device 4 based on the model data (three-dimensional position information) of CG of the tank T, 
and specifically corresponds in CG field of the tank T is made to scan sequentially. That is, the 
distance data to the actual realistic object equivalent to the position of each pixel of CG is 
incorporated one by one, and the successive approximation is carried out to the corresponding 
distance data to each portion of the front face D of CG. And CG (same blue as CG background 
was used) image which masked the CG field X of the tank T judged that the above-mentioned 
realistic object exists in a near side (experiencing person I side) from the construction 
predetermined position of a tank in the field of CG of the above-mentioned tank T is created 
( drawin g 5). 

The landscape image A of an experiencing person's sight line direction acquired by (4) and (1) 
(drawing 2) and CG image ( drawin g 5) to which masking was added by (3) are compounded. That 
is, by the image compositing device 6, the masking portion (blue portion) of CG image (dr awing 5 ) 
is treated as transparent, the landscape image A is compounded, and the image composing Z like 
drawing 6 is displayed on the display 7. Namely, the object (the left-hand side of drawi ng 2 and 
the building a of 5 part, the building b of drawing 2 ) located behind the tank T hides by CG of the 
tank T, The image composing Z compounded so that the object ( drawing 2, the building c of 5, 
the trees d) located in the near side of the tank T might hide a part of CG of the tank T will be 
displayed on the display 7. 

[000 7] Therefore, the scene before and behind the structure which is due to be built, and this 
structure can be compounded with reality, and it comes to be able to perform a real virtual 
experience (viewing). Since he is trying to display continuously the picture ( drawing 2 , 3, 5, 6) 
which processed above-mentioned (1) - (4) to the display 7 of the experiencing person head on 
real time, it comes to be able to perform the virtual experience which has a real feeling there. 
Since it ends with comparison of only a realistic object located in actual corresponding in outline 
of CG scene within the limits when performing distance comparison of the realistic object in a 
scene, and CG, it is not necessary to perform distance measurement of the whole scene, 
distance measurement and the throughput of distance comparison are reduced, and processing 
efficiency is good. 

[0008]While calculating the position of light sources, such as the sun, and giving the shadow S1 
to CG of a structure by inputting date information into the personal computer 5 based on the 
position of said light source, If it is made to change the shadow S1 with change of time, a real 
feeling can be taken out to CG of a structure and CG can be brought close to the actual thing. 
While attaching shade and shadow (shade of a shadow) also in the portion to which light strikes 
upon CG of the structure in consideration of the surroundings lump of light by acquiring 
illuminance information with an illuminometer etc. and inputting this illuminance information into 
the personal computer 5, If it is made to change these shade and shadow with change of 
illumination, a real feeling can be taken out by CG of a structure and CG of a structure can be 
brought more close to the actual thing. In this case, what is necessary is to input beforehand the 
model data of a shadow or shade and shadow to various structures into the personal computer 
5, and just to process change of the shadow by change of date information and illuminance 
information, or shade and shadow by software. Since a virtual experience can be carried out still 
with reality according to the above, still more realistic examination can be performed. 
[0009]Although the data of a picture signal and a sight line direction, a position, etc. is always 
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acquired and he is trying to display a picture on the display 7 continuously above in real time, 
when required, data is acquired once and, of course, it may be made to display a picture each 
time. 

[0010]Without forming the depth sensing device 4 for exclusive use, as shown in drawing 7 , the 
two image photographing apparatus 1A and 1B may be prepared, and it may constitute so that 
the azimuth difference of two pictures may detect depth. In this case, the depth calculation 
software by image processing is built into the personal computer 5A. 

[001 1] As the position sensing device 2 f a followed type total station other than a GPS device, 
etc. can also be used. A marker etc. are fixed to the described image photographing instrument 1 
or the display 7, and it may be made for calculation to detect the present virtual experience 
person's sight line direction from the physical relationship of the marker within a picture by 
reading into the personal computer 5 by making it into picture information, without using the 
direction finding system 3 for exclusive use. As the image compositing device 6, use TEROPPA 
which treats the specified color (monochrome) as transparent besides a chroma keyer, and 
compounds CG of the personal computer 5 to a video signal, or r Or it may be made to combine 
two pictures by the same processing as a chroma keyer or TEROPPA using the video input 
output function added to the personal computer 5. As the display 7, a notebook sized personal 
computer etc. may be used as FMD (face mount display device) or a movable display besides 
HMD. It may be made to transmit a picture and a direction, a position, and depth information 
(data) by radio other than transmission by a cable as a signal-transmission method. It may be 
made to transmit the above-mentioned data digitally using cable LAN or wireless LAN. It is not 
necessary to necessarily attach the image photographing apparatus 1 (1A, 1B), the direction 
finding system 3, the position sensing device 2, the depth sensing device 4, and the display 7 to 
a head. For example, these may all be made to unite with a personal computer, and it is good 
only also considering an image photographing apparatus as a different body. What is necessary is 
in short, just to be able to detect direction of an image photographing apparatus. A CCD camera 
etc. may be attached to a telescope or binoculars and an image photographing apparatus may be 
constituted. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 

[Brief Description of the Drawings] 

[Drawing 1] It is a block linebiock diagram showing the composition of the device by the 
embodiment of the invention 1. 

[Drawing 2] It is an explanatory view for explaining operation of the device by Embodiment 1. 
[Drawjng 3]It is an explanatory view for explaining operation of the device by Embodiment 1. 
[Drawing 4]It is an explanatory view for explaining operation of the device by Embodiment 1. 
CDravying_5]It is an explanatory view for explaining operation of the device by Embodiment 1. 
[Drawin g 6] It ts an explanatory view for explaining operation of the device by Embodiment 1. 
[Dr awing 7] It is a block linebiock diagram showing the composition of the device by other 
embodiments of this invention. 
[Description of Notations] 

1 An image photographing apparatus and 2 [ A personal computer (control means) and 6 / An 
image compositing device and 7 / A display and A / A landscape image, a B CG image, and C / 
A background and Z / Image composing and S1 / Shadow. ] A position sensing device and 3 A 
direction finding system and 4 A depth sensing device and 5 



[Translation done.] 



http://www4.ipdl.inpit.go jp/cgi-bin/tran_web_cgi„eije?atw_u=http%3A%2F%2Fwww4.i... 2008/03/25 



JP,2001-101252,A [DRAWINGS] 



1/3 ^— v 



* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

LThis document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3. In the drawings, any words are not translated. 



DRAWINGS 



[Draw ing 1] 




[Drawin g 3] 
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[Drawing 7] 
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